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Abstract 



We present an analytic method for computing the moments of a 
sum of independent and identically distributed random variables. The 
limiting behavior of these sums is very important to statistical theory, 
and the moment expressions that we derive allow for it to be studied 
relatively easily. We show this by presenting a new proof of the central 
limit theorem and several other convergence results. 
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1 Introduction 

Sums of independent and identically distributed random variables often ap- 
pear in statistics and probability. Their major role in statistics is to provide a 
mathematical model for the estimation of a population mean from a sample. 
We write such sums as 

n 

Sn = ^ ^ -^ii 
i=l 

where Xi,X2, . . . ,Xn are independent and identically distributed {iid) ran- 
dom variables distributed as a random variable X, where E{X) — /i and 
var(X) = cr^. The limiting behavior of these sums is very significant. For 
example, the central limit theorem says that under certain conditions 

Sn-nn 

as n — )■ oo, where Z ~ A'"(0, 1) and ^ denotes weak convergence (convergence 
in distribution). Because of this kind of limiting behavior, sums of iid random 
variables form the basis of asymptotically normal statistical estimation and 
hypothesis testing. 

Probabilists have been interested in the moments of sums of random 
variables since the early part of last century. Khinchine's 1923 paper appears 
to make the first significant contribution to this problem [1]. It provides 
inequalities for the moments of a sum of Bernoulli random variables. In 1970, 
Rosenthal generahsed Khinchine's result to the case of positive or mean-zero 
random variables [2] . Further refinements to these bounds have been made by 
Latala and Hitczenko, Montogomery-Smith and Oleszkiewiez in more recent 
times [3, 4, 5]. Nonetheless, there appears to be no equalities available for 
the case of a sum of iid random variables. Amongst other uses, expressions 
for the moments should allow for the limiting behavior of Sn to be studied 
relatively easily. 

This article presents an analytic method for computing each moment of 
a sum of iid random variables. As an application, we will take limits of these 
expressions and obtain the central limit theorem and some other results. 
These results are probabily easy to obtain by other methods, however they 
do not appear to have been highlighted in the literature. We prove a limit 
theorem for the case where X has an asymmetric distribution about (i.e., 
P(-oo < X < -c) ^ P{c < X < oo)), namely 
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where is a random variable with a unit-step distribution, 




We also obtain a generalisation of the weak law of large numbers, 



lim P{\SP/rf - > e) = 0, 



n- 



for all e > and p = 1,2,..., and a generalisation of the strong law of large 
numbers for when X has a symmetric distribution about = 0, 



where — > indicates almost sure convergence. The conditions under which 
these results hold are discussed in the main text. 

Section 2 provides a formula for the moments and its proof, and also shows 
how the formula is applied. Section 3 studies the limits of these formulas 
section 4 presents some final remarks. 

2 Formula for the moments 

We present the formula for the moments as follows. Fix ap e {1,2,...} and 
define 



[ r + sH \-t 

The pth moment of Sn — X]"=i Xi is 




E{X^)E{X^) ■ ■ ■ E{X') : r, s, . . . , i e {1, 2, . . . , p} 





(1) 



qi&QP 



where, for = E{Xp^)E{Xp^) 



E{XP"^), 



1 n\ p\ 



(2) 



Gi = 



hiy. ■■■lh\{n- m)\pi\p2\ ■ ■ -p, 
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In Equation (1), h is the number of distinct constants in the sequence 
{Pi!P2! • • • iPm}) h the number of elements equal to the first constant, I2 
the number equal to the second constant, . . ., and Ih the number equal to 
the hth. constant (e.g., for the sequence {1, 1, 1, 1}, /i = 1 and li = 4, and for 
{1,2,2,2}, /i = 2, /i = 1, and k = 3). Note that E{Sp) < 00 if an only if 
< 00 for all positive integers a <p. 
For example, the third moment works out to be 



E(Sl) = (1--^:!L^^)e{x')+(^^^^^)e(x')e(x) 

^ \l\{n-l)\'i\) ^ ^ Vl'l'(^-2)!2!l!y ^ > ^ > 

+ (^(n-3)!l!l!l!)^^^^' 

= nE{X^) + 3n(n - 1)E{X^)E{X) + n{n - l)(n - 2)E{Xf 

The proof of Equations (1) and (2) is a lengthly application of various 
combinatorial formulas. 

Proof. The pth moment of Sn is 

n n n 

i=l j=l k=l 
p sums 

Because Xi, . . . , X„ are iid random variables, each term in the sum can be 
factored into the form 

E{XP^)E{XP^)---E{XP''), 

where Pi,P2, ■ ■ ■ ,Pn are positive integers that sum to p. Equation (3) can 
therefore be written as Equation (1). To determine the constants ai, 02, . . . 
fix a positive value of p and positive non-zero integers pi,p2, ■ ■ ■ , Pm such that 
Pi + P2 + ■ ■ ■ + Pm = P- For each choice oi i,j, . . . ,k e {1,2,..., n}, with 
i ^ j ^, . . . ^ k, define the collection 



all unique permutations of the sequence 



Q ^ ) Xi, Xi, . . . Xi, Xj, Xj, . . . Xj, . . . , Xk, Xk, . . .Xk 



PI objects p2 objects Pm objects 
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Each of the sequences in jp2,...,fep^ corresponds to an expectation of the 
form E{Xf^Xf ■ ■■X^"'). These are each equivalent to 

= E{XP')E{XP^)E{XP"') 

The subscript m on is only for bookkeeping purposes. The number of 
elements in Cj^^ jp2,...,fep^ is the number of ways in which q^ can be created 
from pi copies of Xj, p2 copies oi Xj, . . ., copies of X^. For any choice of 
i,j,...,k, this number is 

p\ 

The number of elements in the union 

ij,...,k e {l,...,n} 

is the number of ways can be created from the pi copies of Xj, p2 copies 
of Xj, . . ., pm copies of Xfc, n choices of i, n — 1 choices of j, . . ., n — (m — 1) 
choices of k. This gives the identity 

«- = l2^....,pJ- (6) 

If there are no equalities between the constants pi,p2 - ■ ■ ■ .Pm, then the col- 
lections {Cjp^ are mutually exclusive and so |Qp^^...^p^| is given by 
Equation (4) times the number of collections in the union, namely 

p\ 

nin — 1) ■ ■ ■in — im — 1)) — ; — ; r. 

If there are equalities between pi,P2, ■ ■ - Pm, then some of the collections in 
{Cip^,ip2,-,>^Pm} equivalent. For example, if pi = p2 = Ps, then 

p 



p 




_ p 




p 

Cpj ,6p2 i<^P3 I- 


■y^Pm 


_ p 

5Cp2 1&P3 5- 


■y^Pm 


_ p 

bpi ,0'P2 )Cp3 ,■ 


•y^Pm 
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In the above equalities, the indices given by the eUipses and kp^ do not 
change. Furthermore, if we also had ^4=^5=^6= P7 in addition to 
Pi = P2 = Ps, then we could also write an additional 4! equivalent collections 
for each of the 3! equivalent collections given above, leading to 4!3! equivalent 
collections in the union in Equation (5). In general, if there are h distinct 
constants in the sequence {pi,P2, ■ ■ - Pm}, with li of the elements equal to one 
of these constants, I2 equal to another of these constants, ■ ■ ■ , Ih equal to the 
remaining of these constants, then /i!/2! ■ ■ ■ 4' of the collections in the union 
in Equation (5) are equivalent. The number of elements in Q^^ for the 
general case is therefore found by taking the mutually exclusive result given 
above and dividing through by liU2\ ■ ■ • Ih^-, namely 

\OP I- ^ ^' P- 

Equation (6) then gives Equations (1) and (2). 

□ 



3 Application: Limit theorems for Sn 

The moment equations (1) and (2) are convenient for investigating the be- 
havior of Sn as n — > (X). We can do this with the method of moments: Let 
Fn be the distribution function of Sn and F a distribution function with mo- 
ments mi, 1712, ■ ■ ■■ Assume that F is determined completely by its moments. 
If E{S^) rrik for every A; > 0, then F^ ^ F (see e.g., [6]). 

3.1 The classical central limit theorem 

The classical central limit theorem is particularly easy to derive from the 
moment equations. Without loss of generality we will assume that X has a 
symmetric distribution about 0. Equations (1) and (2) simplify significantly 
in this case because all terms involving odd moments of X are identically 
zero. Moreover, all odd moments of Sn are also identically zero. This can be 
seen from Equation (3). For odd p, at least one of pi, p2, ■ ■ ■, Pn will be odd, 
and so each term in Equation (3) will contain at least one odd moment of X 
and will be zero. The second, fourth and sixth moments for the symmetric 
case work out to be 
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(7) 



E{S^) = n//4 + 3n(n - 1)(7^ (8) 

= n/X6 + 15n{n - l)fX4cr^ + 15n{n -l){n- 2)a^ (9) 

where ji^ = E{X^) and o"^ = E{X'^). After some manipulations, the mo- 
ments of Sn/ {cy/n) take on the form 

where A;!! = k{k — 2)(A; — 4) • • • 1 is the double factorial and (p — 1)!! the pth 
moment of a standard normal random variable. /9„(p) can be regarded as a 
non-normal 'correction term' for the pth moment. While there appears to be 
no general formula for the correction terms, the first few are 

/3n(2) = (11) 

M^) = i - 3) (12) 

In general, f3n{p) ^ for all p > 0, and so E{SP/{nP/^aP)) {p - 1)!! 
completely. The classical central limit theorem then follows from the fact 
that the normal distribution is completely determined by its moments. 

The above assumes that all moments of the random variable X are finite. 
While this covers many important cases, other proofs of the central limit 
theorem do not make this assumption and are less restrictive. 

3.2 Convergence of a sum of asymmetric random vari- 
ables 

Now we consider the case where X has an asymmetric distribution about 0. 
Under this assumption, the odd moments of Sn are no longer zero and there 
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are additional terms in the expressions for the even moments. The first three 
moments are 



E{Sn) = flfl (14) 

E{Sl) = 711^2 + n{n-l)i^^ (15) 

E{Sf,) = nn3 + 3n(n - l)n2H + n{n - l)(n - 2)//, (16) 
and after some manipulations the moments of Sn/ n work out to be 

where Q;„(p) is another correction term. The first three correction terms are 

«n(l) = 



q;„(2) = i (//2 - //^) 
n 



q;„(3) = ^ (//3 - 3/X2Ai + + - (3)U2)U - 3yU^) 
In the general case, we have an{p) for all p > 0, and hence 

E ^ (18) 

The sequence of moments ^, /x^, /x^, . . . determine a probability distribution 
function with a unit-step at /x, namely 



P{H^ < h) 



h < n 

1 /i > // 



This can be checked directly by expanding the characteristic function of H^j, 
into its Taylor's series and making the substitution E{H^) = [J^. 

We have therefore proven the theorem mentioned in the introduction, 
namely 
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Our proof has again required all moments of X to be finite. Loosely speaking, 
Equation (19) says that the probability density of Sn/n concentrates into a 
delta-like point mass as n — )■ oo. This kind of behavior is apparent from 
other asymptotic results as well. For example, it is well-known (and easy to 
show from Equations (14) - (16)) that for very large n Sn/n is approximately 
N{iJ,,a'^/n). This result suggests that as n — >■ oo, the probability density of 
Sn/n 'converges to a delta function' centered at fx. The novelty of Equation 
(19) is that it puts this concept on precise mathematical footing. 



3.3 Generalisations of the laws of large numbers 

Equation (18) in the previous section says that for asymmetric X the pth 
moment of Sn/n converges to fi^. This is also true for symmetric X, in 
which /i = 0. This can be seen from Equations (7) - (9), which show that 
the highest power of n that appears in the numerator of E{S^) is p/2, and 
so for large n 

E {Si/nP) = O ^ 0. (20) 

We will now prove a generalisation of the weak law of large numbers, namely 

lim P(|5^/nf -/x^l > e) = (21) 

n— >oo 

for all e > 0. The usual weak law is the case p — 1. As with the usual law, 
the proof is based on the Markov inequality: 

P {\Sl/n^ - iJ>\ > e) < j,E{{Sl/n^ - fx^f). (22) 

For the case of symmetric X, fi = and so by Equation (20) the right-hand 
side of Equation (22) goes to zero. For the asymmetric case, 

E((S^JnP - ix^f) = EiSl^/n-"^) + /.^^ - 2ix^E{Sl/nP) ^ 

by Equation (18). Taking the limit of Equation (22) then completes the 
proof. 

A generalisation of the strong law of large numbers can be proven for the 
case where X is symmetric. From Equations (7) - (9) we can see that 
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p 

m=l 

where Ci, C2, . . ., Cp are constants and C = max^=i^2,...,p {cto}- Summing 
Equation (22) over n therefore gives 

00 

Y.P{\Slln^\>e) 

n=l 



By the Borel-CanteUi lemma, 

P {u that are in infinitely many {\S^/n^\ > e}) = 0, (23) 

and so there exists an uq such that for n > no, P (|5'^/n^| < e) = 1. Letting 
e ^ 0, we find that 

S'Jn^ ^ 0, (24) 

where — > indicates almost sure convergence. It does not appear possible to 
apply this method to the case of asymmetric X. 

4 Final remarks 

We have presented an analytic method of computing the moments of a sum 
of iid random variables and used it to derive the central limit theorem and 
several other new limiting results. The word 'method' should be emphasised 
because Equations (1) and (2) only give a systematic way of calculating the 
moments, rather than general formula for the moments explicitly. Indeed, 
from Equations (7) to (9) and Equations (14) to (16), the moments do not 
appear to possess any general structure. The method looks useful for nu- 
merical computation of higher moments, although we have not explored this 
possiblity here. 

Outside of statistics, the major application of a sum of iid random vari- 
ables is in the study of random walks, which play a fundamental role in 



< 



e 
C 



n=l 

00 



< 00 



n=l 
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stochastic process theory and statistical physics (see e.g., [7, 8]). We have 
discussed continuous time random walks in a physical context elsewhere [9, 
10, 11]. 
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